In a cluster-based data service, a shared cryptographic key is very useful in that data encrypted by one node in the cluster can always be decrypted by other nodes in the same cluster. In order to ensure the shared key's confidentiality, a general practice is to use a master cryptographic key to wrap or encrypt the shared key so that the shared key is only persisted in wrapped or encrypted form. The master key is usually stored and managed by a key management system (KMS), which is an industry standard solution.
When the shared key is needed, each node can retrieve the master key from the KMS and use the master key to decrypt the wrapped shared key. Once the shared key is extracted, the extracted shared key can only be kept in memory for security considerations, and thus, cannot be in persisted form.
Due to this limitation, when a node in the cluster reboots, the shared key in memory is no longer available. Thus, the node has to connect to the KMS to get master key to decrypt the wrapped shared key, which is stored in persisted form. Unfortunately, if the KMS is not available at that time, the node will have no way to decrypt the wrapped shared key to retrieve the shared key.
One solution to continue to operate a cluster with rebooting nodes and an unavailable KMS would be to request the master key from other nodes in the cluster. However, there would have to be an authentication mechanism for the other nodes to know whether the requesting node belongs to the cluster. Such authentication mechanism would be sophisticated to implement, and may add significant complexity to the system.
Throughout the description, similar reference numbers may be used to identify similar elements.